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Executive Summary 


In late September 2020, Abt along with technology partner Bdna, were commissioned to evaluate the operation 
and effectiveness of the COVIDSafe app and the National COVIDSafe Datastore. The evaluation would inform a 
six-monthly report to be tabled in parliament under Section 94ZA of the Privacy Amendment (Public Health 
Contact Information) Act 2020. 


A mixed-methods approach was used drawing on public health evaluation and technology review methodology 
to examine available evidence on the appropriateness, effectiveness and efficiency of COVIDSafe. The 
evaluation focuses on the three states with the highest number of COVID-19 cases in the country: MevwSouth 
Wales, Victoria and Queensland. 


Appropriateness 


At the time of COVIDSafe development, there was an increasing recognition that to keeo pace with a novel virus 
of such rapid spread the well-proven system of standard contact tracing would berefit from further 
strengthening by leveraging technological innovations. 


Stakeholders agreed that the intended objective of the app which focused om nroviding safeguards to open the 
economy were beneficial. All stakeholders acknowledge the challenges at/tri2time of the app development, 
which involved producing a new digital contact tracing tool, for a new virus, and in a context of evolving 
epidemiological and technological evidence. To do it in a matter of wexs and under intense public scrutiny to 
safeguard both public health priorities and individual privacy rights«vas a commendable achievement. 


That developing COVIDSafe was the right thing to do is reflecteu! by the high evidence-based score given to the 
app by both the Institute of Public Affairs and Per Capita analysis of quality of emergency decision-making in the 
country. Both studies found COVIDSafe to apply best-practice and along with the JobKeeper scheme, the app 
topped the ranks of emergency decision-making in.both studies. 


From a technology perspective, the COVIDSafe app was the correct tool to employ based on the parameters of 
knowledge and capabilities at the time of abo iaunch. Many of the international contact tracing apps, such as 
Singapore’s TraceTogether, similarly utilisee/3luetooth Low Energy (BLE) to capture digital ‘handshakes’ 
between mobile devices. The extent o1.the limitations of utilising BLE technology (discussed further in the 
report) to perform contact tracing-*o7the COVID-19 situation were not known at the time of app launch 
because this particular use of th@+technology did not yet exist. Developing a BLE solution was also especially 
attractive at the time becaus2/user privacy was paramount, so privacy intrusive alternatives (such as 
geolocation trackers) were considered but a decision was made that these alternatives were not suitable to 
protect user privacy. 


Effectiveness 


Digital Transformation Agency (DTA) data (for the period 18 March to 13 October 2020) shows a large number 
of people /7.222,061) have voluntarily installed the COVIDSafe app and registered it. Most users of COVIDSafe 
reside in NSW (33%), VIC (29%), and QLD (18%) with the remainder of the states making up the last 20%. 


During*the second half of 2020, NSW provided ‘ideal’ conditions to evaluate the effectiveness of the app (an 
open economy with moderate to high potential transmission). In NSW, the total number of true close contacts 
identified by the app represents 1.3% of the total pool of close contacts identified in NSW, (tolerance 0.8% — 
1.8% under different scenarios). In Victoria, state data shows that out of the 12,609 cases with COVIDSafe 
information recorded, 1,833 cases (15%) reported using the app versus 10,776 cases (85%) reporting no app- 
usage. Victoria also provided data on the number of close contacts identified by the sub-sample of app-using 
cases (9,402 close contacts recorded for 1,833 app-using cases). In all Victorian cases they were all recorded as 
individuals already identified by standard contact tracing with no novel contacts identified through the app. 


During the second wave of the pandemic, Queensland closed state borders and progressively relaxed social and 
mobility restrictions for its residents. The state showed very low levels of community transmission with only 60 
confirmed cases locally or inter-state acquired between 1 May and 15 November 2020. The contact tracing 
team identified 4,273 close contacts, that is an average of 71 per case. Less than five cases were reported by 


Final Report - Evaluation of the Operation and Effectiveness of COVIDSafe and the National COVIDSafe Datastore — March 2021 

Department of Health - No part of this report may be changed or altered without the express permission of Abt Associates a1 
OFFICIAL 

FOI 2326 Document 2 5 of 35 


OFFICIAL 


Queensland Health as using the app in the data provided and no novel contacts were identified using the app. 
For the sub-sample of cases over 18 years of age (48), this represents a proportion of app-using cases in the 
range of two and eight percent. 


COVIDSafe Limitations 


The COVIDSafe app utilises BLE technology, cryptography, operates independently on a user’s device and does 
not require any integration with other apps to perform its function. However, our technology review suggests 
there are some important technology constraints that might have contributed to the level of app effectivenes» 


The COVIDSafe app relies on users selecting the right phone settings for both iOS and Android devices. Gn 
Android and iOS devices, the COVIDSafe app is not able to function properly (i.e. register handshakes) witnout 
having Bluetooth enabled, this is also true of the COVID-19 Google and Apple Exposure Notificationyrsamework 
(ENF) and so is a challenge for all Bluetooth contact tracing apps. 


For Android phones, location services also need to be enabled and for all phones the apge ther needs to be 
open and active or running in the background. This means there is a level of user compliance required as a pre- 
requisite for the app to work effectively and this has not always occurred and is andinherent risk for the effective 


operation of COVIDSafe. s47C, s47E(d) 
s47C, s47E(d) 


The iOS operating system has several inherent limitations including a sécUrity feature implemented by Apple 
which prevents apps from constantly running Bluetooth in the backg.ouind if that data is going to be moved off 
device. 

s47C, s47E(d) 


s47C, s47E(d) Abt also observed citriing our interviews that state contact tracing teams 
identified very high-risk settings (indoor venues, ssich as restaurants, bars and nightclubs as well as large 
unstructured outdoor events such as music.festivals, food festivals, carnivals, and other un-ticketed spectator 
events) as a key priority gap in standard contact tracing that would need the support of digital tracing tools, 
such as attendance registration apps or COVIDSafe like technology options. However, in its current form, 
COVIDSafe is not able to be used effectively for these purposes due to the identified constraints discussed in 
this report. 


The app has identified less then two percent of the total pool of close contacts identified in NSW and no new 
contacts were identifiedsbs_COVIDSafe in Victoria or Queensland between March and November 2020. 
COVIDSafe has technology constraints, complications with manufacturer phone settings, and there are 
limitations to its useniiv high-risk settings. Overall, the state contact tracers interviewed for this report have 
suggested it hasssot been a highly effective complementary tool for their state contact tracing systems. 


Efficiency 


During.ciu interviews, the primary users and beneficiaries of COVIDSafe, the state contact tracing teams, have 
noted ‘hat the time required to undertake the various COVIDSafe tasks adds to their workloads without an 
optimisation of benefits. Contact tracers reported that they mainly use COVIDSafe to quality assure manual 
wacing efforts at the end of the standard contact tracing interview. When asked if COVIDSafe could be used 
higher up in the interview process, all states unanimously suggested this was not possible due to a lack of app 
data for the majority of potential contacts. Available data from NSW examined show that out of 205 individuals 
flagged by the app, 61% (126) were ‘false close contacts’, 30% (62) were individuals already identified by 
standard contact tracing and the remaining 8% (17) were novel contacts. These inefficiencies arise because 
while all app potential contacts need to be validated by contact tracers, a significant number of those 
individuals are either ‘false close contacts’ or individuals already identified by standard contact tracing. This is 
compounded by existing processes to access datastore information. 


Due to stringent privacy protections and without a data exchange facility, all information needs to be examined 
on screen by contact tracers and manually entered into the jurisdictional systems or spreadsheets. Contact 
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tracers are not able to download the data, automate or match information with their state systems to analyse 
and validate information in a less cumbersome way. 


In the current environment, where public health units must be ready to rapidly surge their capacity and where it 
is widely acknowledged that experienced contact tracers are at the core of a good system and cannot be easily 
outsourced or secured through short term training, inefficient use of contact tracers’ time is a high cost for the 
system to bear. Since experienced contact tracers are currently in high demand and face scarce supply, any 
additional task required from them that does not deliver substantial benefits carries considerable opportunit/ 
costs. 


Overall, the utilisation of COVIDSafe has resulted in high transaction costs for state contact tracing tearas and 
produced few benefits and efficiencies to the existing contact tracing work-flow. 


Conclusion 


As countries around the world grappled with the dire economic consequences of lockdcowns, there was 
increasing recognition that as economies started opening again, digital tools were required to support contact 
tracers and effectively manage the pandemic. Almost a year later, the s47C call for end-to-end contact 
tracing that never falls behind reinforces this vision of leveraging technology,<apabilities to support an effective 
test and trace system. 


From a technology perspective, the COVIDSafe app was the correct teeito employ based on the knowledge and 
capabilities at the time of app launch. Many of the international contact tracing apps, utilised Bluetooth 
technology to capture digital ‘handshakes’ between mobile dévices. 


While the tool used appropriate technology and there have=zeen several notable improvements to the app and 
datastore since it was launched, from the perspective of svate contact tracers who are the beneficiaries or 
customers of the app, there are limitations in both the effectiveness and efficiency of COVIDSafe for contact 
tracing efforts. 


We acknowledge that even with a vaccine, C@ViB-19 will remain a challenge for some time and in this context, 
we believe it is crucial for all state and commonwealth stakeholders to explore options for enhancing 
COVIDSafe. It is particularly important ta, the seven million Australians that have downloaded the app, to 
acknowledge the performance barrié?s, that are limiting the effectiveness and efficiency of COVIDSafe. 


We hope these evaluation findirigs-can be used to inform discussions with state contact tracers on future 
improvements to COVIDSafe, We further advise that noticeable improvements to COVIDSafe should occur and 
be demonstrated throughtae continued involvement of states and territory public health officials in user testing 
(the workflow of the datastore, integrations with state contact tracing systems and the utility of the app), to 
ensure continued sugport of the app. 
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Introduction 


With the COVID-19 pandemic creating havoc around the world, in April 2020 the Department of Home Affairs 
was considering developing a contact tracing app, based on one already in use by the Singaporean government 
called TraceTogether. The Commonwealth government decided that to be successful in the Australian context, 
the app needed to be voluntary; have a strict focus on health; reassure the public that it would not track the 
location of citizens; and have robust privacy and security safeguards built in. The policy side of the app was 
transferred to the Department of Health (DoH) from the Department of Home Affairs and the Digital 
Transformation Agency (DTA) provided high level product development guidance. The DTA team was’task2d to 
develop a new digital contact tracing tool in a matter of weeks and under intense public scrutiny thet would 
support contact tracing efforts as economic restrictions were lifted. This culminated in the developrrient of the 
COVIDSafe App. 


At the time of launch the intended objective of COVIDSafe was to support Public Health-lnits (PHUs) as a 
complementary tool to their manual tracing processes. The rationale was that as leckaawns were coming to an 
end and as people resumed their normally active lives, confirmed cases might not-easily be able to recall or even 
know the people they were subsequently in close contact with. The app was lasincned on 26 April 2020 and the 
Prime Minister’s press release! detailed all the privacy considerations that wauid allow Australians to download 
the app confidently in the knowledge that the government would not use the app to track their location and 
that their personal data was securely protected. 


In late September 2020 Abt along with technology partner Bdria were commissioned to evaluate the operation 
and effectiveness of the COVIDSafe app and the National COViOSafe Datastore. The evaluation would inform a 
six-monthly report to be tabled in parliament under Sectiow94ZA of the Privacy Amendment (Public Health 
Contact Information) Act 2020. 


A mixed-methods approach was used drawing on. sstelic health evaluation and technology review methodology 
to examine available evidence on the appropriaieriess, effectiveness and efficiency of COVIDSafe. Our 
evaluation focuses on three states, New South Wales, Victoria and Queensland, chosen in consultation with the 
Department of Health as the states with tiie iighest number of COVID-19 cases in the country and with PHUs 
who most use the datastore. 


Out of scope aspects of the evaluation included other health aspects of the response to the COVID-19 pandemic 
as well as privacy issues that havepbeen evaluated separately. The review team consulted with state and 


territory public health authar‘ties who are responsible for contact tracing s47C, s47E(d) 
s47C, s47E(d) 


s47C, s47E(4) Additionally, only publicly available updates to COVIDSate, such as 
those discussed in the media have been examined and illustrate the Commonwealth’s commitment to 
continuous improvement?. 


the findings presented in this report are subject to the limitations and assumptions of the data, including 
differences identified between Commonwealth and State data sources. The evaluators have sought to identify 
and minimise, to the greatest extent possible, these limitations. While the analysis is firmly based on best 
practice and has been subjected to rigorous quality assurance procedures, ongoing developments and 
improvements in data may potentially produce different results in different components of the analysis. 


Contextual Background for COVIDSafe Development 


Throughout the COVID-19 pandemic, governments across the world have been grappling with difficult choices in 
the middle of uncertainty. Under normal circumstances, we deal with uncertainty by gathering additional 
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information, but these were not normal circumstances. Constantly emerging new knowledge about the 
epidemiology of the virus’ and lack of consensus amongst experts as to which specific strategies would deliver 
the best balance of health and economic outcomes: have been the ‘new norm’ under COVID-19. 


Policymakers have had no option but to make decisions in the context of uncertainty, without knowing the level 
of outcomes that could be expected and the associated probabilities that would occur. Unfortunately, not even 
the best fine-tuned and state of the art epidemiological and behavioural models could have addressed each and 
every single question that needed an answer. The recent discussions about vaccines‘ is just one example of tive 
enormous challenge that this unchartered territory poses to even the most judicious policy-makers set on 
following the science and working in close collaboration with teams of epidemiologists, modellers and other 
scientists. 


At the time of COVIDSafe development, there was increasing recognition that to keep pace with@nsvel virus of 
such rapid spread as mobility and social restrictions were lifted, the well-proven system of standard contact 
tracing would benefit from further strengthening by leveraging technological innovations.”, Government 
officials’ concerns about the enormity of the risk of opening the economy without safeguards, such as those 
provided by digital contact tracing supports, reflected broader concerns shared byacatemics about the risks 
posed by the so-called exit strategies that allow countries to lift economic restrictions such as lockdowns. ® 
Modelling by the University of Melbourne’ showed that an unmitigated pandemic would rapidly overwhelm the 
health system capacity in Australia and although the economy has performed much better than expected, this is 
the largest shock to economic growth since the 1930s and hence the desit2410 open the economy as soon as 
feasible. In the next decade it is expected that Australian governmentwest as a result of the pandemic might be 
over half a trillion dollars higher than it would otherwise have been? 


Early papers based on the best available modelling parametersvat the time suggested that delays on standard 
contact tracing could be substantially reduced with app-besed contact tracing, provided of course their 
modelling assumptions that had never been tested wot'd ‘hold in real life settings 4, 2. That those modelling 
assumptions were perhaps excessively optimistic was proven by an August systematic review that found no 
empirical evidence of the impact of automated contact tracing on metrics such as number of contacts identified 
and transmission reduction. However, those movwelling studies were the best and perhaps the only available 
evidence at the time. Along the same lines, at the time of launch, the key concern for policymakers was to 
support contact tracers when cases failed ta-recall or did not know the names and contact details of those 
people they had been in close contact with (for example on public transport), which would target encounters 
that put people at risk of infection-dve'to physical proximity. However, in the coming months strong evidence 
started emerging that the airbosietransmission characteristic of COVID-19 was remarkably efficient and a 
priority concern in terms of peiiaemic management as it appeared to be the only plausible explanation for 
several large outbreaks ari@’ciusters in various enclosed settings across the world, which are not the target of 
proximity apps like COViIDSafe™. 


The success of Aust#aiia’s management of the pandemic looms particularly large in a world where even 
advanced econcmies continue to struggle to contain both the virus spread and economic damage." It has also 
obscured the enormity of the opportunity costs of not doing the right thing**, which includes investing in 
innovatioiys, such as digital contact tracing tools, that showed promising potential to support opening the 
econgmcand relaxing restrictions safely. 


Afterfactoring in the novelty of the virus and its evolving evidence base; the uncertainty under which decisions 
had to be made; as well as the vast human, social and economic costs associated with the pandemic, investing 
in innovations such as COVIDSafe aimed at supporting contact tracing systems across the country was not only 
the right thing to do, but it also made good economic sense, if only because the costs of ‘doing nothing’, 
mounting to billions and even trillions of dollars, were just too large to ignore. 


‘Doing the right thing’ in such an uncertain environment involved delivering innovative interventions, such as 
COVIDSafe that showed promising potential to deliver large benefits by contributing to the largely successful 
pandemic response efforts of government. That developing COVIDSafe was the right thing to do is reflected by 
the high evidence-based score given to the app by both the Institute of Public Affairs and Per Capita analysis of 
quality of emergency decision-making in the country. Both studies found COVIDSafe to apply best-practice and 
along with the JobKeeper scheme, the app topped the ranks of emergency decision-making in both studies.?’,8 
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COVIDSafe investments, approximately $16 million in direct costs such as app development and publicity (i.e. 
excluding staff costs), have been dwarfed by other COVID-19 government investments, such as the 1° $1.1 billion 
spent since the start of the pandemic in Medicare-subsidised pathology testing for COVID-19” and over $3.3 
billion directed towards securing a vaccine.” With app costs representing less than 0.004% of just these two 
items, there is not even a suggestion that at the time of the app development taking the risk of investing in a 
technological innovation such as COVIDSafe was a reckless decision that carried an important opportunity cost 
for the Commonwealth budget. 


COVIDSafe was based on the Singaporean open-source BlueTrace/Opentrace protocol which uses BLE 
technology to exchange digital handshakes. These digital handshakes were considered to be a proxy forclese 
contact or physical proximity between two individuals. 


Importantly, downloading and using COVIDSafe is also voluntary and a similar privacy-by-design “ivodel was 
adopted with a range of privacy and security safeguards built in. This was in keeping with they;cemimonwealth 
Government’s requirements to not have location tracking included in the app. 


The COVIDSafe website provides detailed information to the public on the various aspects of the app and gives 
due priority to issues related to user’s privacy, including information on the Privacy: Amendment passed by 
Parliament on 14 May 2020, the COVIDSafe Privacy Policy” available in 63 languazes and the Privacy Impact 
Assessment was commissioned during the first half of 2020. 


Privacy issues are outside the scope of this review (albeit it is a factor ofutuke), but suffice to say that available 
evidence suggests that the robust privacy and security safeguards asseciaced with the app encouraged a large 
number of Australians to download the app. 


A July 2020 study on data trust and data privacy in the COVID*%9 period showed that public trust in COVIDSafe 
associated organisations with regards to data was found tore a strong predictor of the probability of individuals 
downloading COVIDSafe. 23 Contrary to this finding, but“eintorcing the importance of privacy to Australian 
citizens, the Australian Bureau of Statistics Household Iriwpacts of COVID-19 Survey (June 2020), found that 24% 
of adults who did not download the app reported/Coricerns for privacy as the primary reason. * 


Methods 


Our evaluation used a mixed-methousand multi-disciplinary approach blending perspectives from public health 
evaluation and technology reviews*to examine available quantitative and qualitative evidence on the 
appropriateness, effectiveness and efficiency of the COVIDSafe app and the National COVIDSafe Datastore. A 
high-level technology review was undertaken according to the Request for Quotation which excluded 


COVIDSafe technologicanalternatives and recommendations for future innovations. s47C, s47E(d) 
s47C, s47E(d) 


For our quaiitative component, information was primarily gathered through virtual interviews, focus groups and 
key infornvant interviews with Commonwealth program partners and state public health offices. Average 
intervievduration was of 60-90 minutes. Discussions were audio-recorded (with consent) and recordings and 
notes were managed as commercial-in-confidence. We spoke to 19 individual stakeholders and conducted 14 
separate interviews (See Appendix 2). 


Primary data collection from app users was out of scope and secondary data provided by the DoH, DTA and the 
three jurisdictions has been used for quantitative analysis. 


Queensland provided tabulated information on their number of cases and close contacts with basic 
demographic breakdown, but without disaggregation by app-usage status due to small numbers. Victoria and 
NSW provided information on cases, close contacts and app-using cases. They use different proxies to capture 
app-using cases and provide different levels of granularity, so their individual metrics are not directly 
comparable. 
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NSW (Kirby Institute, UNSW & NSW Health) reported in their interim summary evaluation report** of 
COVIDSafe, key metrics for cases over 13 years of age that reported using the app and had it activated at least 
for some time during their infectious period, but without demographic breakdown. 


Victoria provided tabulated data on the total number of cases and the demographic breakdown for cases with 
app-using information recorded. However, no demographic breakdown for the total sample of cases was 
available. They also provided information on the number of novel contacts identified by the app for the 
corresponding sub-sample, but without detailed information on the total number of individuals flagged by the 
app and whether they were true or false close contacts. 


State datasets were triangulated with the information directly provided to the DoH for the second half of 2020, 
compiled for the researchers by the DoH team in a spreadsheet. We complemented this analysis withen 
examination of the DTA COVIDSafe report provided in the last week of the evaluation. 


The findings presented in this report are subject to the limitations and assumptions of the data, including 
differences between Commonwealth and state data sources. The evaluators have sought tovidentify and 
minimise, to the greatest extent possible, these limitations which could be from the rrethods used to define and 
collect the data. While the analysis is firmly based on best practice and has been suxiected to rigorous quality 
assurance procedures, ongoing developments and improvements in data may poteritially produce different 
results in different components of the analysis. 


Appropriateness 


In this section we examine the operational performance of CCY|IDSafe within the parameters of knowledge and 
capabilities at the time of launch. We first set the policy ana@technology context at the time of app development 
which provides the framework for our review of the exteni.to which the app and the datastore operated as 
intended/designed and whether the technology supyvoried the operational performance of COVIDSafe. A brief 
discussion of the Singaporean experience is included in Appendix 1. 


COVIDSafe Design and Operatieh 


COVIDSafe aims to enhance and supplement the manual process of tracing people who have been in close 
contact with someone with COVID-19%,, including to verify and prioritise cases. COVIDSafe is a standalone 
contact tracing app that utilises'!uetooth Low Energy (BLE) technology to record Bluetooth “handshakes” with 
other nearby COVIDSafe enaljleadevices. 


The Bluetooth handshakexexchange a payload that contains information such as device details, Bluetooth 
received signal strength (RSSI) and datetime. 


To further increase‘privacy around user identity, an update included functionality to randomly generate a new 
temporary ID (Temp ID) every few minutes and exchange this in the Bluetooth handshakes instead of a user ID. 
The only wav*to decrypt the Temp ID and view the original user ID is once the data is voluntarily passed into the 
data store 


A clase contact is registered by the app when there has been a sequence of Bluetooth handshakes lasting longer 
thar? 15 minutes from the first to last handshake and where there is a medium to high probability that at least 
eine of the handshakes is within 1.5 metres. 


As per the COVIDSafe Health Officials Training 26(V15.4 PowerPoint), once an individual is confirmed to have 
COVID-19 (now a case) and consents to having their close contact data uploaded for contact tracing purposes, 
the process of uploading the case’s close contact data by Public Health Officials is as follows: 

e Log into the COVIDSafe Health Portal; 

e Search for the case using their phone number; 

e Generate a PIN and send it to the case’s phone; 


e Data is uploaded from case’s phone; 
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e Review the close contacts of the case; 
e Produce a final list of close contacts; and 


e Contact or pass the list of close contacts onto the contact tracing team. 


From a technology perspective, the COVIDSafe app was the correct tool to employ based on the parameters of 
knowledge and capabilities at the time of app launch. Many of the international contact tracing apps, such as 
Singapore’s TraceTogether, similarly utilised BLE to capture digital ‘handshakes’ between mobile devices. The 
extent of the limitations of utilising BLE technology (discussed further below) to perform contact tracing for the 
COVID-19 situation were not known at the time of app launch because this particular use of the technology uid 
not yet exist. Developing a BLE solution was also especially attractive at the time because user privacywias 
paramount, so privacy intrusive alternatives (such as a geolocation tracker) were rejected. 


One Commonwealth employee noted that you needed the privacy controls to get the public tac,euup, but the 
controls could have undermined the delivery of the app. The COVIDSafe app was designed with user privacy and 
security as the highest priorities (enshrined in specific legislation) and these have been adnered to. The data is 
stored locally on the smartphone, the data can only be uploaded with explicit permission from the user and the 
only identifiable information captured is the user’s phone number (as the name c2n‘be a pseudonym). This 
stored information is all end-to-end encrypted and cannot be read unless it’s passed into the National 
COVIDSafe Data Store. COVIDSafe was updated on 7 September 2020, primefi!v for security reasons (replay 
attacks) but also had the additional benefit of further improving user privacy.” 


In terms of the security of the National COVIDSafe Data Store, the CG'ViUsafe project has also complied with 
best practices by hosting the data store on an Amazon Web Services {AWS) cloud platform. AWS has military 
grade security in place and as per the contract between the Australian Government and Amazon, AWS has no 
access to the encrypted information from COVIDSafe and the data cannot leave Australia.” It is a punishable 
criminal offence to hold COVIDSafe data outside of Australiay disclose COVIDSafe app information to any 
persons outside of Australia or use the information for any other purpose than state/territory health officials 
performing contact tracing”. 


As previously mentioned, COVIDSafe was desigineu to make users’ mobile devices proxies for face-to-face 
contact and capture these encounters to assist In manual contact tracing processes. To that end, the app 
accomplishes what it was intended to withiiv the parameters of technology, although as we discuss below there 
were technical limitations that constrained its effectiveness. 


The National COVIDSaf@Data Store 


The National COVIDSafe Diata’Store was designed to store case contact data once a user is confirmed to be 
COVID-19 positive and has consented to have their contact data uploaded. It has also been demonstrated that 
the National COVIDSare Data Store is capable of receiving and uploading encounter data from users. However, 
there were some\issues with accessing data as state contact tracers mentioned that sometimes ‘there was no 
data to uplaad’=t was unclear whether this was because of an issue with the datastore, an issue with the app 
itself or beriavioural and based around how users use the app on their device. Other issues with the public 
health portal included the complexity of the information provided to contact tracers, mostly during the period 
immediately after COVIDSafe launch which was subsequently improved. All state stakeholders noted the 
responsiveness of the DTA and DoH COVIDSafe teams, which undertook research with contact tracing teams in 
Victoria, NSW, Queensland and Western Australia to identify opportunities for improving their interface with 
the health portal. They also acknowledged that substantial improvements were made in how the information 
is filtered and presented to PHUs, making it a more user-friendly process. This engagement should be continued 
to make further enhancements that align with the needs of the contact tracers and encourage ongoing use. 


To safeguard users’ privacy and comply with legislation, COVIDSafe was designed and built with stringent data 
access protocols in place to prevent automatic data exchanges. Data exchanges require the states to ensure 
they have adequate privacy protection legislation and processes in place and, as a result, need to be manually 
performed. To further illustrate the extent of the constraints, even for the purposes of this evaluation, de- 
identifiable data for basic variables such as the demographic profile of users could not be secured. In interviews 
with the state contact tracers, they all agreed that these data access protocols resulted in a cumbersome and 
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inefficient process for them to access the information, which involves eyeing data on screen and then manually 
entering any information into the contact tracing team’s management system or a spreadsheet. 


According to state contact tracers, they ‘eye ball’ the web portal and write down the names of potential close 
contacts, and then cross reference the data with what they have in their system and in other places to see 
whether it's likely that what 1) they already know about it, so no further action is taken and 2) if it is a new 
contact, to talk to the individual and understand the nature of the interaction they've had with the case. 


Our technology review did not include any analysis of the National COVIDSafe Data Store algorithms such as 
those used to classify ‘close contacts’ visible to PHUs (stable and sporadic) nor any others. With this cavect in 
mind, we conclude that the technology behind the COVIDSafe app and data store is appropriate to reccrdjand 
provide a list of digital encounters for an individual confirmed to have COVID-19 with these digital envounters 
serving as a proxy for personal contact, so long as there are no issues with the BLE technology, ofactier 
implementation issues as are discussed further in the effectiveness and efficiency sections. 


In order to maximise the app uptake, the Australian government made the decision for the GOVIDSafe app to 
not capture location data and this decision is likely supported by the privacy/location*soncerns demonstrated 
through the s47C, s47E(d) report and ABS survey. Based on the decision to not utilise location tracking, the 
choice to utilise BLE technology is appropriate. As intended, the COVIDSafe Data Store has strict privacy and 
data security protocols that safeguards users’ data even at the expense of autoniatic data exchange with PHUs. 


COVIDSafe Coverage 


Available evidence shows a large number of people have voluntayi installed the COVIDSafe app and registered. 
However, during the reporting period there was not data providea that suggests how many are continuing to 
actively use it. DTA data (18 March to 13 October 2020) shswed that registered users stood at 7,122,061. If we 
assume only one registration this represents 28% of theAwsiralian population (of 26 million Australians, 91% 
have smartphones)?},2. 


State User registrations as a perceritze of total 
NSW 33% 

VIC 29% 

QLD 18% 

WA 10% 

SA 7% 

TAS 2% 

ACT <1% 

NT <1% 


App usage as Gefined by consumer research 
s47C, s47E(d) 
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Effectiveness 


The primary purpose of COVIDSafe was to support contact tracing teams when State ane Territory economies 
started opening after the first pandemic wave. Abt examined available evidence, primarily of quantitative 
nature, to analyse the effectiveness of COVIDSafe support to contact tracing teanss-across the three states; 
Queensland, NSW and Victoria. 


Two key sets of data were available for the evaluation. One provided by ttiestates and a second one provided 
by DTA. We had access to tabulated data on key indicators for Queensiaiid, Victoria and DTA. NSW provided key 
indicators estimated in the ‘Interim summary evaluation of the effectiveness and utility of the COVIDSafe app as 
an integrated contact tracing tool for the COVID-19 response’ unaertaken by the Kirby Institute, UNSW Sydney 
in collaboration with the NSW Ministry of Health. We examined relevant metrics from both data sources for 
NSW and Victoria. For Queensland, numbers were too sma‘tor our analysis, so we have provided a high-level 
description of the number of cases and close contacts. 


New South Wales 


Since the launch of the app, a second wave et infections and an economy that remained open with relatively 
moderate social restrictions resulted in-a relatively high potential of virus transmission in NSW. 2° University of 
Melbourne modelling showed that i?ithe month of October and with the virus still circulating in the community, 
NSW with an open economy had _&ijansmission potential of 0.93 vs. 0.56 in Victoria which was still under stage 
4 lockdown. The virus transmissio.i potential in NSW illustrates the heavy demands imposed on contact tracing 
and testing systems to continue managing the epidemic in an open economy, while also providing the ideal 
setting for examining the «ffectiveness of the app. 


NSW interim evaiuation data 
1. State data irom the NSW interim evaluation summarised in 


Table 1 be!ow show that from 4 May to 30 October 2020: 


2.\.NSW recorded 620 cases acquired locally or interstate aged 13 years or older and 137 (22%) had the app 
activated for at least part of their infectious period; 


3. App data was obtained for 67% of app-using cases (92). The majority of cases for which data were not 
accessed (31 out of 45) had been in quarantine during the entire infectious period; 


4. NSW Health followed up a total of approximately 25,386 close contacts, that is an average of 41 close 
contacts per case; 


5. COVIDSafe data obtained for 92 cases revealed 205 potential close contacts, that is, on average over two 
individuals flagged by the app per case; 


6. During case interviews, contact tracers validated the app data, which resulted in 126 out of 205 potential 
close contacts (61%) not meeting the epidemiological and clinical criteria of close contact (referred to as 
‘false close contacts’ by contact tracers); 
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7. Asaresult 79 individuals flagged by the app in NSW were assessed by contact tracers as true close 
contacts (39% of all app potential contacts). They represent 0.3% of the state pool of close contacts 
(25,386), a proxy for the positivity rate of the app, that is the ability to flag individuals who meet the 
clinical and epidemiological criteria of close contact; 


8. With 79 true close contacts, the app identified 0.86 close contacts per case for whom data were accessed 
vs. 41 close contacts per case for the overall contact tracing sample; and 


9. Out of those 79 app true close contacts 22%, that is 17 individuals, correspond to contacts not identified 
by standard contact tracing, a figure also quoted in previous parliament discussions °°. 


Table 1 - NSW Cases and Close Contacts (4 May - 30 October 2020)* 


NSW Cases and Close Contacts Number (#) and 
(04 May - 30 October) percentage (%) 


Cases 620 
Close Contacts identified/followed up manually 25,386 
Close contacts per case a 
Cases with app activated 137 


- As % of all cases 


Cases with app data accessed 92 


- As%ofall cases 
- As %of cases with app activated 


Potential contacts flagged by the app 205 
App false close contacts 126 
; 19 
-  As%of app potentia’ ¢:,.acts GL 
App true close contacts 79 
0, 
-  As%of appiratential contacts oan 
True close conté 21s per case 0.86 
True Close-Contacts as % of all NSW close contacts 0.31% 
rr eee 
[% 
Apa Novel close contacts 17 
22% 
- As%of t | 
s % of app true close contacts 0.07% 
- As % of all NSW close contacts 
App contacts also identified by standard contact tracing 62 
78% 


-  As%of app true close contacts 


*Locally or interstate acquired confirmed cases aged 13 years or older 


Source: Evaluating the effectiveness and utility of the COVIDSafe app as an integrated contact tracing tool for the COVID-19 
response — Interim results from NSW 34 
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DTA COVIDSafe Report NSW 
The DTA COVIDSafe report shows that since 26 April to 27 November 2020 in NSW: 


1. 105 uploads were registered and 85% of those uploads (89) had at least one encounter recorded. 
2. For those 105 encounters, 837 app-identified contacts were recorded, that is, 7.9 contacts per upload. 


As noted in the DTA report, the list of potential contacts are calculated as a ‘count of distinct relationships 
between uploader and contacts’, and correspond to the ‘number of potential close contacts identified when a 
PHO has uploaded the data of a positive case’. 


It can be seen that there are important differences with the indicators provided by the NSW interim “valuation 
report. DTA data was only made available on 18 December 2020 and we were unable to discuss these 
differences with state officials. We only have access to aggregated data, so we are unable to cleaxly establish 
the source of the differences, which would require examining disaggregated information fram both data 
sources. However, with a view to contribute to a future reconciliation of these figures bv. DIA and State 
representatives, we examine below the key differences and explore plausible implications. 


s47C, s47E(d) 
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Victoria 


Inithe second half of 2020, Victoria experienced a second wave of infections with tightened mobility restrictions 
brought about by the state government from mid/late June 2020. 


Melbourne entered a stage 4 lockdown on 6 August 2020 just a few days after daily surges peaked near 700 
cases.?” 


Mobility and social restrictions such as those imposed by Victoria aim at breaking the chain of transmission by 
bringing down contact rates between cases and other individuals. In such environments, people have fewer 
random contacts in close proximity and easily remember those individuals with whom they were in close 
contact and there was consensus in the consultations with state contact tracers that digital tools like COVIDSafe 
were not designed to support contact tracing in this context. With this caveat in mind, we examine below 
aggregated data from both data sources; state and DTA. 
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State Data 
Victoria data in Table 5 show that from 1 May to 15 October 202028 


e Data on app-usage was collected for 67% of all cases acquired locally or interstate (12,609 out of 
18,947). The remaining 6,338 cases (33%) were reported as having missing information. 

e Out of the 12,609 cases with COVIDSafe information recorded, 1,833 cases (15%) reported using the 
app versus 10,776 cases (85%) reporting no app-usage. Similar proportions are observed when the 
sub-sample of cases over 18 years of old is used (16% of app-using cases vs. 84% reporting no us@): 


The state also provided data on the number of close contacts identified by the sub-sample of app-usingscases 
(9,402 close contacts recorded for 1,833 app-using cases)**. 


All contacts were recorded as individuals already identified by standard manual contact tracii\g with no novel 
contacts identified through the app. 


We note that progressively increasing mobility restrictions and social distancing measures were imposed since 
June and it is widely agreed that there is no expectation for the app to demonstrate‘effectiveness in these 
circumstances. 


The state did not record information on the number of individuals flagged 4v.trie app that were found not to 
meet the criteria of close contact. 


Table 5 - Victoria - Case Reporting of COVIDSafe Use (01 May- 31st. October 2020) 


VF 
COVIDSafe Use * Number of Bases | 
o. 
Yes 1,838 “15%) 
No 19,776 (85%) 


Sub-total of Cases with COVIDSafe Use informatian~ | 12,609 


Cases with Missing information 6,338 


Total 18,947 


*Locally or interstate acquired confirtaec»cases; Source Victoria State Government 
2020, Data on COVID-19 Cases, Clcs¢ Contacts and COVIDSafe, provided to the 
evaluation team?® 


DTA COVIDSafe-Report Victoria 
The DTA reportvata for Victoria shows: 


e 625 udleads and 75% with encounters registered (466); and 


e 1,;4é7 potential close contacts were identified, that is 2.7 per upload. 


The)research team do not have state level data on the number of individuals flagged by the app, who were 
reviewed by contact tracers and the proportions that meet the criteria of close contact versus false close 
contact. 


We are thus unable to gauge the extent to which information on close contacts provided by DTA aligns with the 
numbers that contact tracers reviewed and validated on screen. However, we briefly examine below the large 
differences observed in terms of uploads and cases with information on app-usage recorded between the two 
data sources. 


s47C, s47E(d) 
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Queensland 


During the second wave of the pandemic, Queensland closed state borders erid progressively relaxed social and 
mobility restrictions between May and November 2020 for residents w thin the state. The state showed very 
low levels of community transmission with only 60 confirmed caseslecally or inter-state acquired in the period 
1 May to 15 November 2020. 


The contact tracing team identified 4,273 close contacts, an_average of 71 per case.*? However, the sample of 
app-using cases was too small for reporting. 


Indicatively there were less than five cases reported using the app in the data from Queensland. For the sub- 
sample of cases over 18 years of age (48), this represents a proportion of app-using cases in the range of two 
and eight percent. Possible reasons for this inclui!e the number of pre-conditions required for the COVIDSafe to 
work effectively (incl. phone settings) and atvigh proportion of potential contacts not meeting the COVIDSafe 
business rules for data to uploaded to the National Datastore. 


In summary, numbers in Queensland _4ré too small for an analysis of the app effectiveness and the social 
distancing. Finally, mobility restrictions in Victoria are not conducive for examining the effectiveness of a digital 
contact tracing tool aimed at siscorting the identification of individuals in close proximity to the case. 


However, available data f(c¢y NSW suggest limited effectiveness of the app, even after allowing for differences 
in data sources. We explere below some of the potential barriers to app effectiveness. 


Effectiveness Drivers 


At the mostbasic level, the effectiveness of the app is predicated on its ability to identify individuals who meet 
the critetia‘of close contact and according to the Series of National Guidelines (SoNG) for Public health offices, 
“a clesexcontact is defined as requiring”: 


1) face-to-face contact in any setting with a confirmed or probable case, for greater than 15 minutes 
cumulative over the course of a week, in the period extending from 48 hours before onset of symptoms in 
the confirmed or probable case, or 


2) sharing of a closed space with a confirmed or probable case for a prolonged period (e.g. more than 2 hours) 
in the period extending from 48 hours before onset of symptoms in the confirmed or probable case.” 


Note: this report is not designed to provide an extended analysis of the definition of close contact, but rather to 
examine barriers for COVIDSafe to support the identification of these two different types of close contact as 
framed within the SoNG and relevant complimentary state guidelines. 
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Face-to-face contacts 


Face-to-face contacts are the primary target of proximity apps like COVIDSafe. The app’s current business rules 
allow for recording of digital handshakes when other app users are in close proximity (i.e. within 1.5 meters). 

However, encounters between two individuals are only logged if during the last 21 days there has been at least 
one single encounter lasting over 15 minutes, excluding the one-week cumulative rule in the current guidelines. 


We should note that the cumulative rule was introduced in SoNG Version 2.5.1 on 17th of April, just a few davs 
after the announcement that the government was developing COVIDSafe. The COVIDSafe algorithm rules are 
not and were never based on the Communicable Disease Network Australia’s (CDNA) SoNG, but rather those set 
out by the AHPPC. 


Data was not available to examine the extent to which a considerable or relatively minor proporticn et close 
contacts would fall in the app excluded category (i.e. over 15 minutes cumulative over a week Wut Without a 
single 15-minute encounter). 


However, interviewees noted that routine face-to-face contacts meeting the cumulativecriteria are likely to be 

easily remembered by the case during standard contact tracing interviews, and suggested that for them there 

might be limited added value in improving the app business rules by incorporating<tie cumulative criteria. 
s47C, s47E(d) 


Technology review: Bluetooth handshakes as proxy fordace-to-face contacts 


The COVIDSafe app utilises BLE technology, cryptography and oes not require any integration with other apps 
to perform its function. However, several technology related issues might prevent smartphone users from 
installing the app or prevent the installed app from registering handshakes that act as a proxy for face-to-face 
contact. 


The COVIDSafe app relies on users selecting the rigiit phone settings for both iOS and Android devices. On 
Android and iOS devices, the COVIDSafe apps ot able to function properly (i.e. register handshakes) without 
having Bluetooth enabled, this is also trueatthe COVID-19 Google and Apple Exposure Notification Framework 
(ENF) and so is a challenge for all Bluetooth contact tracing apps. 


For Android phones, location services-also need to be enabled and for all phones the app either needs to be 
open and active or running in the ‘sackground. For some users This means there is a level of user compliance 
required as a pre-requisite f9.the app to work effectively and this has not always occurred and is an inherent 
risk for the effective operotion of COVIDSafe. Location services also need to be enabled for Android users 
because Google requires iocation permissions for Android apps to access Bluetooth.” This may also be part of 
the reason that thenuinber of app registrations has not translated into a high number of potential close 
contacts who have COVIDSafe data when contacted by state contact tracing teams, and further investigation on 
this is warrantea. 


However>!eis a very common misconception that the app records location s47C, s47E(d) 

whick yaund that 57% of people surveyed believe the app tracks their location and collects location data. This 
misunderstanding likely resulted in lower engagement with the app, particularly amongst key demographic 
groups such as young people, which s47C, s47E(d) notes is a key barrier to be addressed if more people 
are to be encouraged to actively use the app. 


In order for the COVIDSafe app to function optimally (i.e. registering handshakes), it either needs to be open 
and active or running in the background. On Android devices, the app can be safely closed, and it will continue 
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to register handshakes in the background, as per the permanent notification in the notification tray. This 
permanent notification also informs the user when the app is inactive and handshake registration cannot occur. 


Figure 1 - COVIDSafe active/not active notification on Android devices 


1:17 pm 1:18 pm 
COVIDSafe is active COVIDSafe is not active 


Keep COVIDSafe active when you leave home or are in .. Allow Location permissions for Bluetooth to work 


Notification settings Notification settings 


s47C, s47E(d) 


Technology writer James Vincent” wrote in May 2020 that this issue of Bluetooth broadcasts (in the 
background) was known at the time of the app launch: 


Both Google and Apple restrict how apps can use Bluetooth in iOS and Android. They don’t allow 
developers to constantly broadcast Bluetooth signals, as that sort of background broadcast has been 
exploited in the past for targeted advertising. As others have reported, iOS apps can only send Bluetooth 
signals when the app is running in the foreground. If your iPhone is locked or you’re not looking at the 
app, then there’s no signal. The latest versions of Android have similar restrictions, only allowing 
Bluetooth signals to be sent out for a few minutes after an app has closed. Such restrictions will block 
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devices from pinging one another in close quarters, drastically reducing the effectiveness of any contact- 
tracing app. 


s47C, s47E(d) 


Figure 3 - The DTA’s COVIDSafe Bluetooth Encounter Loggins, Results 


COVIDSafe Bluetooth Encounter Logging Results 
As at 26 May 2020 
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Enolosed settings 


Individuals sharing enclosed spaces are not the primary target of proximity apps like COVIDSafe, but of 
attendance registration or ‘check-in’ apps, which facilitate the recording of people attending public venues such 
as cafes and restaurants® 


Enclosed environments provide a heightened risk due to airborne transmission and various factors including 
poor air ventilation and extended exposure periods. *°,°°. The basic rules of the definition (enclosed space and 
exposure of more than two hours) were current at the time of the app launch, though as late as June 2020, the 
risk of airborne transmission, except for aerosol-generating procedures performed in health care was not 
recognised by most public health organisations, including WHO. Evidence however was mounting that airborne 
transmission in enclosed, crowded environments with inadequate ventilation appeared to be the only plausible 
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explanation for several super-spreading events. This prompted calls for governments around the world to take 
firmer action to prevent virus transmission in these settings, including moving activities outdoors. >4,°2 


Current NSW and Victoria guidelines reflect the heightened risks posed by enclosed spaces.®?,, and ample 
media coverage of large outbreaks linked to this type of setting have shown the Australian public their potential 
for rapid virus transmission®. For contact tracing teams ‘never fall behind’, digital contact tracing supports, 
including contact tracing apps are a must in this type of settings if large numbers of potential contacts are to be 
rapidly identified and quarantined*®. 


Although not the intended target of COVIDSafe, at least in theory, the app could be useful in identifying 
individuals that shared a closed space and in some instances it could even provide useful information for 
contact tracers to unveil unreported exposure sites that the case failed to mention during a standard contact 
tracing interview. This was indeed the rationale for describing validation of close contacts as oneef tne key 
benefits of the app. On the other hand, risk exposure in this type of settings happen even when.contacts 
maintain physical distancing from a case (i.e. over 1.5 metres), resulting in a substantial rumber of exposed 
individuals excluded from COVIDSafe business rules identification. 


Both the potential benefit for COVIDSafe to uncover unreported exposure sites witi,potential to become 
hotspots can be illustrated by the following example which was provided by Commonwealth staff and was 
commonly referred to as the Mounties example in the media‘. A previously «nreported exposure at a venue 
assessed as high risk was identified from COVIDSafe data. In addition to.several close contacts present at the 
venue who were identified directly from the App, 544 close contacts wére,identified from a venue sign-in list 
and requested to self-quarantine. Two of those contacts subsequent tested positive to COVID-19. 


These new cases linked to this venue may have been unveilec’In subsequent interviews, and the failure to 
identify those two close contacts that tested positive could have led to a ‘super-spreader’ event that highlights 
the benefits of COVIDSafe to supplement contact tracing..!’owever, as noted by the s47C, __, effective 
contact tracing in this type of setting requires both locétion data and personal data which is only recorded via 
check-in lists at present. This suggests the app (whica doesn’t record location) may not be as efficient for 
supporting contact tracing in this context, whichxisow a key priority for contact tracing but was never the 
intended purpose of the app. 


Very high-risk environmentsywhere digital tracing tools may yield additional 
benefits 


In September 2020, the Austratieri Health Protection Principal Committee released a statement on very high- 
risk environments which cary the risk of COVID-19 transmission due to a wide range of factors including large 
numbers, crowding and*queuing, surfaces with multiple high touch points, close proximity, loud volume singing, 
multiple venues opetasing at the same time, people visiting multiple venues and people attending from or 
returning to regiaialareas. 5” Very high-risk settings include typical indoor venues discussed above, such as 
restaurants, bars and nightclubs as well as large unstructured outdoor events such as music festivals, food 
festivals, scheolies graduation festivals, carnivals, community sporting events and other un-ticketed spectator 
events. 7nese settings can easily become hotspots for large outbreaks and potential super-spreader events 
wheré many individuals can become infected on one occasion®. s47C, we also 
observed during our interviews, that state contact tracing teams identified enclosed settings as a key priority 
gep in standard contact tracing that would need the support of digital tracing tools, such as attendance 
registration apps or COVIDSafe like technology options. However, in its current form, COVIDSafe is not able to 
be used effectively for these purposes due to the identified constraints already discussed above. 


Along similar lines, the other type of setting in which technology aids with area tracing capability would provide 
additional benefits are unstructured settings such as supermarkets, major retail shops, shopping centres, 
community shared facilities and unstructured community gatherings not easily amenable to attendance 
registration apps. Contact tracers use a combination of approaches including aide-memoire tools as well as 
social and mass media messaging that rely on identification of specific locations and times visited by the case. 
As noted by one of our interviewees: 
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The second piece (for) any technical solution, needs to not only include the potential close contacts, but a 
connection to support jurisdictions with the check-in functionality, like the venue check in, which 
predominantly is QR stuff, but it also allows us to keep a better handle on movements of people in social 
settings and quickly get access to that data if we need to, to 1) talk to them legitimately as close contacts, 
but 2) to also send out broadcast comms and media updates to, keep a track of where people are, if we 
need to get in contact with them quickly (Contact tracing team staff member) 


Finally, s47C, s47E(d) , efficient contact tracing relies on the accurate memory recall o1 
cases and jurisdictions have been relying on information stored on cases’ smartphones like diaries, GPS 
coordinates and time stamps of photos to help refresh their memories”. In theory COVIDSafe could also prove 
beneficial in at least some of these circumstances. However, as noted in DoH training materials, inferimation on 
the phone number and name of individuals identified by the app should not be revealed to the case, which 
makes the task of leveraging the app information for refreshing people’s memories, validation er eliciting new 
information somewhat onerous. 


The app has identified less than two percent of the total pool of close contacts identiried in NSW and no new 
contacts were identified by COVIDSafe in Victoria or Queensland between March aiid November 2020. 
COVIDSafe has technology constraints, complications with manufacturer phone settings, and there are 
limitations to its use in high-risk settings. Overall, the state contact tracers int2rviewed for this report have 
suggested it has not been a highly effective complementary tool for their state contact tracing systems. 


Efficiency 


The rationale s47C, s47E(d) emphasised the importance of rapid contact tracing for bringing a return to 
‘normal’ economic activity as well as the need to reduce tie time elapsing from patient testing to quarantine of 
their close contacts to fewer than 48 hours“. This was a!so reflected in the original intent of the COVIDSafe app, 
which was to contribute to a more rapid, effective nd efficient contact tracing process. 


During our interviews, the primary users of C6ViDSafe, the state contact tracing teams, have noted that the 
time required to undertake the various CQViOSafe tasks adds to their workloads without optimisation of 
benefits. Available data from NSW exanined in the previous chapter, show that out of 205 individuals flagged 
by the app, 61% (126) were ‘false close tontacts’, 30% (62) were individuals already identified by standard 
contact tracing and the remaining-S% (17) were novel contacts. 


So, inefficiencies arise because while all app potential contacts need to be validated by contact tracers, a 
significant number of thos@idividuals are either ‘false close contacts’ or individuals already identified by 
standard contact tracing», This is compounded by existing processes to access datastore information. We 
examine each of thesé issues below in more detail. 


Verifying information on potential close contacts flagged by the app 


During tii2 ‘case interviews, contact tracers ask cases whether they are a COVIDSafe user or not. Contact tracers 
then.braceed to secure consent to upload their data to the datastore and request the pin for uploading. If users 
are quarantine, then this process is not always followed, due to the lack of exposure. According to 
intérviewees, this process usually happens relatively smoothly and without delays, although in some rare 
instances uploading might fail and has to take place at a different time. Once data are uploaded, contact tracers 
must review and validate the app contact information on screen, which involves cross-checking it against the list 
of close contacts secured during the interview and flag those ones listed in the app, but who have not been 
identified by standard contact tracing. 


Contact tracers need to verify information for each individual to establish the circumstances of registered 
encounters and assess whether the person has been at risk of exposure or not. In line with the training 
materials and the COVIDSafe legislation to avoid privacy breaches that can easily happen if contact tracers let 
slip the name of a person that has not yet been provided by the case, extreme care needs to be taken during the 
interview to verify the information of these potential contacts. Contact tracers would for example start 
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enquiring about the whereabouts of the case at the time of the encounter listed by the app and would establish 
if given the circumstances, the individual flagged by the app was at risk of exposure or not. So even though a 
concurrent use of the health portal might in theory be an efficient way of embedding COVIDSafe into existing 
contact tracing processes and leverage COVIDSafe®, a case interview usually needs to happen first and it is used 
to verify if individuals flagged by the app were at risk of exposure, without privacy breaches. 


False close contacts and technology barriers 


Those individuals who are assessed as not being at risk of exposure are classified by state contact tracers .4$ 
‘false close contacts’. In line with the nuances of contact tracing, some instances of ‘false close contacts’ are 
easily verified during the interview as the examples provided by Victoria stakeholders who noted thawa’pattern 
emerged of individuals listed by the app who had registered encounters with the case that lasteafrom late at 
night to early in the morning and who happened to be neighbours living in attached dwellings separated by a 
wall and at no risk of infection. 


In other instances, the validation process is more involved and resource intensive. Fox.eéxaimple, Queensland 
stakeholders described a situation requiring validation of multiple encounters registered by the app that 
happened to take place during a waiting queue for COVID-19. Contact tracers hac to reverse engineer the line- 
up using lab registrations and cross-reference them and they finally establish“a‘that approximately 17 to 18 
individuals flagged by the app were not at risk of exposure since they werewet in close proximity to the case for 
15 minutes or more. 


Our evidence suggests that most ‘false close contacts’ were indivicars'listed by the app as meeting the criteria 
of face-to-face contact as they were likely to be within 1.5 mevers.or the case for 15 minutes or longer, but after 
investigation by contact tracers, most of them were found to be physically separated from the case. 


In addition to false close contacts being physically sepaxatery, other issues are technology related; the first of 

these is that Bluetooth technology is not consistent hetween phones; and the second possible explanation is 

that the Received Signal Strength Indication (RSSIeaix vary substantially depending on a variety of factors. 
s47C, s47E(d) 
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Inefficiencies associated with current processes fei.accessing datastore 
information 


As part of their quality improvement processes for an agile pioduct like COVIDSafe, in August 2020, DTA 
undertook research to better understand how state teams.u:se the COVIDSafe Health Portal to complement 
their existing contact tracing processes. They examinec.ew-contact tracing processes varied across states and 
identified constraints and opportunities for improvement, including those related to how the data are 
presented to contact tracers. State stakeholders neted that substantial improvements have been made to the 
health portal that de-emphasise technology-spécitic content that contact tracers found confusing and facilitate 
the use of the information during interviews®. 


However, the report also noted that the process of accessing the health portal is manual and takes time™, which 
contributes to system inefficiencies. “/ithout a data exchange facility for contact tracers to easily access 
COVIDSafe contact data, all inforriati6n needs to be examined on screen and manually entered into their own 
systems and spreadsheets. Contavt tracers are not able to download the data into a CSV file (due to privacy 
issues which are still being wecked through), which would in their opinion, “allow them to analyse and validate 
information in a less cumbersome way.” 


As one contact trace¢said: 


What weuiiawe most valuable from (COVIDSafe in the future), if you get a list of, by numbers out of the 
app direcily, we cross reference that with a list, that comes out of the case interview, because then we 
coulacross off all the people that we already knew about really quickly. And be left with a subset of 
cgritacts that we had that hadn't emerged through the manual contact tracing process. But...that's not an 
option because of the privacy and security constraints that are implemented with the use of the system. 


(Contact tracing team staff member) 


We do not have data to estimate time spent by contact tracers uploading, analysing and validating the potential 
close contact information provided by COVIDSafe, but the current COVIDSafe interface adds another layer of 
tasks to contact tracing teams who need to manually validate and analyse information on individuals flagged by 
the app, drawing on data that they have already secured through their standard contact tracing. The contact 
tracers interviewed each gave their own estimates as to the amount of extra time they spent accessing and 
analysing the data from the National COVIDSafe Data Store; some of these estimates were as long as 2 hours. 


In the current environment, where PHUs must be ready to rapidly surge their capacity and where it is widely 
acknowledged that experienced contact tracers are at the core of a good system and cannot be easily 
outsourced or secured through short term training“, inefficient use of contact tracers’ time is a high cost for the 
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system to bear. Since experienced contact tracers are currently in high demand and face scarce supply, any 
additional task required from them that does not deliver substantial benefits carries considerable opportunity 
costs, as some of the following state contact tracers (who are the ultimate customers of the app) have 
described: 


And there's a lot of work to squeeze out that value out of what's there, which detracts from a scarce 
resource that could be doing boring, but legitimate manual contact tracing work to identify high-risk 
household contacts and to, you know, get the case of support they need and to do all of the other stuff 
that we're doing. 


(Contact tracing team staff member) 


| think it's most useful in settings where there's no registration, possibly like bus or a subway'sr a train 
carriage would be effective....but at this point it hasn't ... identified a lone person on a seet behind 
someone ona bus or a subway carriage or something 


(Contact tracing team staff member) 


Overall, the utilisation of COVIDSafe has resulted in high transaction costs for state contact tracing teams and 
produced few benefits and efficiencies to the existing contact tracing work-flow. 


Conclusions 


As countries around the world grappled with the dire economicconsequences of lockdowns, there was 
increasing recognition that as economies started opening.again, digital tools were required to support contact 


tracers and effectively manage the pandemic®. s47C, s#7E(d) 
s47C, s47E(d) 


As our technology review indicates, based ef.thie parameters of knowledge and capabilities at the time of app 
launch, it is believed that the COVIDSafe apr was the correct tool to employ. Many of the international contact 
tracing apps, such as Singapore’s Trace Together, utilised BLE to capture digital ‘handshakes’ between mobile 
devices. The limitations of utilising-BY& technology to perform contact tracing were not known at the time of 
app launch because this particuiex_app of the technology did not yet exist. Utilising a BLE solution was also 
especially attractive at the tiroesbecause user privacy was paramount, so privacy intrusive alternatives such as a 
geolocation tracker were (ejected. 


Stakeholders agreed that the intended objective of the app which focused on providing safeguards to open the 
economy were beneficial. All stakeholders acknowledge the challenges at the time of the app development, 
which involved }iducing a new digital contact tracing tool, for a new virus, and in a context of evolving 
epidemiological and technological evidence. To do it in a matter of weeks and under intense public scrutiny to 
safeguard both public health priorities and individual privacy rights was a commendable achievement. 


We have found limitations in both the effectiveness and efficiency of COVIDSafe for contact tracing efforts and 
acknowledge that even with a vaccine, COVID-19 will remain a challenge for some time. In this context, we 
believe it is crucial for all state and commonwealth stakeholders to explore options for enhancing COVIDSafe, 
particularly in regards to the seven million Australians that have downloaded the app, while recognising the real 
performance barriers that are limiting the effectiveness and efficiency of COVIDSafe. 


We hope these evaluation findings can be used to inform discussions with state contact tracers on future 
improvements to COVIDSafe. We further advise that noticeable improvements to COVIDSafe should occur and 
be demonstrated through the continued involvement of states and territory public health officials in user testing 
(the workflow of the datastore, integrations with state contact tracing systems and the utility of the app), to 
ensure continued support of the app. 
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Appendix 1: Singapore’s TraceTogether and SafeEntry 


Early in the pandemic, Singapore was at the forefront of contact tracing apps. The Singaporean government 
developed an open-source app protocol called OpenTrace/BlueTrace which was eventually developed into the 
country’s national contact tracing app, TraceTogether on which the COVIDSafe app was based. The 
development of a COVID Bluetooth app was pioneered by Singapore with their TraceTogether app, released ov 
20 March 2020.* If a person was diagnosed with COVID-19, they could permit the Ministry of Health access to 
their TraceTogether Bluetooth proximity data to support the contact tracing of those who were in closé.cantact 
with the diagnosed app user. Developers were open about the technology not replacing manual ceneact tracing 
but rather complement - a combination of centralised contact tracing and follow-up, and this coula never 
replace human involvement.” Their adoption of a ‘hybrid model’ of a decentralised and centralised approach 
was highlighted as being specifically built for Singapore and what was felt to work best farmtrveir population and 
economy.® 


In the first 24 hours, TraceTogether app received half a million downloads. A morthlater, an estimated 1.1 
million users had downloaded the app, with an adoption rate of 20% of the papulation.* The TraceTogether 
technology was quickly adopted around the world by countries such as thé enited States, South Korea, India 
and Israel.”? However, Singapore changed direction shortly after and beza); adopting more traditional public 
health strategies. The download rate plateaued despite the demonstiated efforts to advertise the technology as 
being a ‘privacy-by-design’ model (designed with data storagevorithe users phone for 21 days and then 
deleted).” This stall in downloads brought to the forefront the concerns around the app’s privacy and people's 
motivations, along with other complaints such as batterv.a¢ainage.”2 Despite these concerns around app privacy, 
the TraceTogether app does not record or send any user ;ocation data”. The COVIDSafe technical review team 
has directly examined the source code for BlueTrase;'OpentTrace, the code base for TraceTogether, and can 
confirm that no location data is recorded or sent: 


TraceTogether released an update in July 2620 that the Singaporean Government claimed had resolved the iOS 
Bluetooth issues. TraceTogether also developed a standalone beacon device, called the TraceTogether token, 
that performs and manages the con‘aect tracing, without needing a smartphone. The main objective of this 
TraceTogether token was to enahie the 5% of the population without smartphones, such as young children and 
the elderly (one of the most atrisk demographics) to participate in contact tracing. As of 25 October 2020, more 
than 400,000 TraceTogether tokens have been collected which, when combined with the TraceTogether app 
downloads, totals about 50% of the Singaporean population participating in contact tracing.” 


Singapore also adonied other means of tracing potential contacts, particularly where significant time was being 
spent in public.claces. This was predominately achieved via the scanning of an individuals’ national identity, 
employmert;or work permit card supported by the Government and an app called SafeEntry.” In addition, 
Singapore.tvas also been reported as using media such as video recordings and surveillance of public locations to 
cross check contract tracing lists — a practice which would not be permissible under Australian privacy law.” As 
of September 2020, TraceTogether tokens can be used for check-in to SafeEntry, so users only need to scan 
their tokens without needing to manually enter their details. 


The Singaporean government has announced that as of December 2020, either TraceTogether or the 
TraceTogether token will be mandatory to gain access to public venues such as restaurants, workplaces, schools 
and shopping malls. Because of this, more than 4.2 million people (78 percent of the Singaporean population) 
are now using either the TraceTogether app or token and this number is only expected to increase as 
distribution of the tokens resumes.” The required use of TraceTogether to access public venues has been 
pushed back due to distribution shortages of the TraceTogether token and is expected to be officially instated 
sometime in January 2021.” 
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Appendix 2: Stakeholder Engagement 


Key Stakeholder Key Staff / Areas / Departments Consultation Type Sample 

Interviewed 
State/Territory State public health officers from the Key Informant Interviews 8 
Governments respective Department of Health. Group interviews 


Public health offices Based on the advice of DoH focusing on Key Informant Interviews 34 
several Public health offices to conduct 
case studies and scenario plans to 
understand contact tracing efforts 


Group interviews 


Commonwealth Department of Health Key Informant Interviews 8 


Agencies Digital Transformation Agency Group interviews 
Document review 


Subtotal Total number of group and key informant 19 
interviews (14) and participants (19) 


Appendix 3: Data sets reviewed 


Datasets used in this evaluation as well as key reports were.orovided directly by DTA or DoH, jurisdictional 
public health offices, and the following kinds of informatioand metrics were referenced: 

e DTAsummary of COVIDSafe metrics 

e = Registration data 

e Information on changes to the 15 min rwieaigorithm 

e Enterprise testing and assurance 


e Enhancements to COVIDSafe 
s47C, s47E(d) 


84/U, S4/E(d) 


e Qualitative data fromystate contact tracing teams 
e = Training presentatiois for state contact tracing teams 


e =©NSW, VIC QLD“sase numbers, cases reported using app, confirmed cases, close contacts 


www.abtassociates.com 
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